1. Field of the Invention
The present invention pertains generally to electrical computers and more particularly to systems and methods for managing and distributing test data in a software development environment.
2. Description of Related Art
When a programmer, or a team of programmers, develops a software program or suite of programs, the software must be tested to verify its proper operation. One type of testing is called "regression testing", which is intended to exhaustively verify the proper operation of every portion of the software. Ideally, every line of software code and every possible execution path in the software is tested to achieve a regression test "coverage" of 100%. To achieve or approach such complete test coverage, regression testing generally requires input data representing one or more test case scenarios that are designed to exercise various aspects of the software. In combination, the test cases are generally intended to verify proper operation of all functional aspects of the software, with particular emphasis on crucial functionality, boundary conditions, exception handling, and specific real world conditions.
When an individual software component is being developed as part of a large software system, such as a Computer-Aided-Engineering (CAE) system or a telecommunications system, it is also advantageous to test each software component using the same or similar test cases so as to effect consistent testing throughout the software system. For example, in a CAE system for processing microelectronic circuit designs, a programmer may choose to test the software on a variety of designs. A typical combination of test cases might include an "exhaustive" design having a comprehensive selection of logic cells, interconnections, and operating conditions. Another test case might include an artificially large and complex design to test the functional capacity of the software. A third test case might include a design that includes a set of logic cells and combinational logic directed to specifically test particular functionality in the software. Test cases may also include non-design data, such as technology files, delay files, and parameter files, required by the particular software under test. Such designs and input files are typically used to test other programs in the CAE system as well.
A single test case may also comprise multiple designs. For example, a CAE program may consolidate multiple designs into a single design and, therefore, require multiple designs as inputs. Generally, test case designs are developed specifically for the purpose of testing the system's software, although real world designs are also frequently used to test such software. A final test case set is often determined at the discretion of the programmer or programming team based on software test requirements and specific target customers needs.
Furthermore, software testing typically involves verification of the software's proper operation against a set of expected results. Generally, each input test case is associated with a particular set of expected output data and, therefore, the input and expected output data are preferably managed together. As such, a programmer may wish to include both input test data and expected output data in a single test case.
In addition to multiple data sets, or test cases, a programmer may wish to organize test data in multiple versions. Whether the versions apply to individual test cases or to individual elements of test cases, the need to manage versions of test data further compounds the complexity of the testing process. Moreover, when test data is created, modified and used by multiple programmers testing individual software programs, the complexity of managing random access to a complex set of test data is amplified.
Another type of software testing is called "integration testing". In a CAE system, for example, where several software programs are used together to process circuit designs, each software component is required to function properly on the output of a previous software component in the engineering process and is required to generate proper output for a subsequent software component in the process. Accordingly, such interrelated software is typically tested sequentially, one software component receiving input from the previous software component to verify proper functionality and inter-compatibility at each stage in the process. Other types of testing, including user interface testing, endurance testing, and simulations, may also require management of input data and expected results.